Quick, Roll a Hallucination Check!